Simplify approach for creating dag run and task spans by dstandish · Pull Request #62554 · apache/airflow

dstandish · 2026-02-27T04:53:38Z

Replace otel spans tracking implementation with something much simpler

rip out existing spans
add back the dag run span
add back a task span (this is handled in the runner)
- the detailed task spans will be done in a followup
~~rip out Trace, _TraceMeta, DebugTrace "classes"~~ --> later
~~remove the "Tracer" protocol and EmptyTrace~~ --> later
- we don't need to keep them for backcompat because they were never really used~
- EmptyTrace was returned when otel not enabled. but it should be fine to lean on the normal behavior of otel to create non-recording spans when otel not enabled, and the updates to OtelTrace should accomplish this
  deprecate the OtelTrace class
- update its methods so that it does as little as possible but doesn't break dags for any unfortunate souls that decided to use it
re-implement the configuration bit which looks at our deprecated otel config vars and the env vars supported by otel

airflow-core/src/airflow/executors/base_executor.py

shared/observability/src/airflow_shared/observability/traces/otel_tracer.py

task-sdk/src/airflow/sdk/observability/trace.py

airflow-core/src/airflow/observability/traces/__init__.py

airflow-core/src/airflow/settings.py

xBis7

@dstandish Great job! I left some comments but generally it looks good and I think it can be converted to an actual PR.

Additionally, I tested it manually and I ran a dag with long timeouts. I didn't see any issues.

airflow-core/src/airflow/observability/traces/__init__.py

airflow-core/tests/integration/otel/test_otel.py

airflow-core/src/airflow/executors/workloads/task.py

integration tests fix static check test fix? try fix int test fix test fix tests fix tests fix integration test fix tests pre-commit

This reverts commit f453689.

uranusjr · 2026-03-10T03:41:25Z

airflow-core/src/airflow/config_templates/config.yml

      type: string
      example: ~
      default: "False"
+    otel_task_runner_span_flush_timeout_millis:


In another config we use the full word “milliseconds” so this should match. That’s the only place we use milliseconds though; elsewhere we always use seconds instead.

millis vs msec no opinion on (Other than the "go duration" mode of 1s/300ms would be nice here but out of scope for this PR).

As for seconds vs milliseconds here: It appears the millis is only an artifact of the Python API, and this isn't commonly set as an environment variable in OTEL installs, so we can choose what makes most sense for us, so we could have this as float number of seconds and convert to the format otel wants.

WDYT @uranusjr ?

ashb · 2026-03-10T10:20:54Z

airflow-core/src/airflow/config_templates/config.yml

      type: string
      example: ~
      default: "False"
+    otel_task_runner_span_flush_timeout_millis:


It's too late to change this now, but the otel_ prefix on all of this is needless :(

ashb · 2026-03-10T10:26:08Z

airflow-core/src/airflow/config_templates/config.yml

+        Timeout in milliseconds to wait for the OpenTelemetry span exporter to flush pending spans
+        when a task runner process exits. If the exporter does not finish within this time, any
+        buffered spans may be dropped.
+      version_added: 3.1.0


Suggested change

version_added: 3.1.0

version_added: 3.2.0

ashb · 2026-03-10T10:36:48Z

airflow-core/src/airflow/models/dagrun.py

+            status_code = StatusCode.OK if state == DagRunState.SUCCESS else StatusCode.ERROR
+            span.set_status(status_code)
+            span.end()


@nickstenning @xBis7 Question on the otel/tracing side. What happens if we never emit this DagRun span, or don't emit it for for days?

It's possible the dag run is left in the running state-- one case when this can happen is that the dag run is started and then the user goes and pauses the Dag in the UI/API, which will mean the state of the dag run is never finiallized.

(I don't think we can do much about this, just might need to document it)

If I understand correctly, we will only see the task spans. The dag run will need to finish so that the span is created and exported. If that happens, then the dag run span will be the parent of the task spans but until then we will only see the children.

ashb · 2026-03-10T10:42:05Z

task-sdk/src/airflow/sdk/execution_time/task_runner.py

+        TraceContextTextMapPropagator().extract(msg.ti.context_carrier) if msg.ti.context_carrier else None
+    )
+    ti = msg.ti
+    span_name = f"task_run.{ti.task_id}"


@nickstenning Should span names be namespaced too?

ashb · 2026-03-10T10:42:42Z

task-sdk/src/airflow/sdk/execution_time/task_runner.py

+            from airflow.sdk.configuration import conf
+
+            timeout_millis = conf.getint(
+                "traces", "otel_task_runner_span_flush_timeout_millis", fallback=30000


Does this config need to be specific to task_runner, or could it apply to all places where we flush before exit?

ashb

A couple of small questions, but looks good to me.

xBis7 · 2026-03-10T11:25:03Z

@dstandish @ashb Please take a look at this #62554 (comment)

I can't unresolve the conversation and the comments are hidden.

Apart from that, it looks good to me.

boring-cyborg bot added area:DAG-processing area:Executors-core LocalExecutor & SequentialExecutor area:Scheduler including HA (high availability) scheduler area:task-sdk area:Triggerer labels Feb 27, 2026

xBis7 reviewed Feb 27, 2026

View reviewed changes

airflow-core/src/airflow/executors/base_executor.py Outdated Show resolved Hide resolved

airflow-core/src/airflow/executors/base_executor.py Outdated Show resolved Hide resolved

xBis7 mentioned this pull request Feb 27, 2026

Cleanup all debug traces #62572

Merged

2 tasks

xBis7 reviewed Feb 27, 2026

View reviewed changes

shared/observability/src/airflow_shared/observability/traces/otel_tracer.py Outdated Show resolved Hide resolved

task-sdk/src/airflow/sdk/observability/trace.py Outdated Show resolved Hide resolved

airflow-core/src/airflow/observability/traces/__init__.py Outdated Show resolved Hide resolved

dstandish force-pushed the minimize-otel-code branch 3 times, most recently from 74b3942 to 236e5ee Compare March 3, 2026 16:40

dstandish changed the title ~~WIP - Minimize otel code~~ Simplify approach for creating dag run and task spans Mar 3, 2026

ashb reviewed Mar 3, 2026

View reviewed changes

airflow-core/src/airflow/settings.py Outdated Show resolved Hide resolved

dstandish force-pushed the minimize-otel-code branch 2 times, most recently from a1cb165 to a85b5eb Compare March 5, 2026 03:29

xBis7 reviewed Mar 5, 2026

View reviewed changes

dstandish force-pushed the minimize-otel-code branch from a7766b3 to 7e0a97e Compare March 5, 2026 22:25

dstandish marked this pull request as ready for review March 5, 2026 22:47

dstandish requested review from XD-DENG, amoghrajesh, bugraoz93, dheerajturaga, gopidesupavan, hussein-awala, jason810496, jscheffl, kaxil, o-nikolas, pierrejeambrun and potiuk as code owners March 5, 2026 22:47

dstandish added 21 commits March 9, 2026 20:37

make configure_otel a no-op when otel_on is off

3b3a75d

review nits

41e0df3

simplify id gen

d5a5558

can set the context carrier earlier

0552f8f

fix test

20e78cc

fixes

8e3608d

type hint for carrier

9a84061

Integration test fix

69cfa73

integration tests fix static check test fix? try fix int test fix test fix tests fix tests fix integration test fix tests pre-commit

fixes

81a40d9

don't override span id

cdfc661

Revert "don't override span id"

773d324

This reverts commit f453689.

use non-recording span to avoid memory leak, and move to helper function

a90b062

fixes

477aa33

fix off by 1

e1e55f7

update span names

d7fbb4a

add config for flush timeout millis

504ac3e

tweak attr names

58279c6

fix tests

c33482a

fix test

76f122b

spelling

b909fdd

christos suggestion

b4166e7

dstandish force-pushed the minimize-otel-code branch from d90deb0 to b4166e7 Compare March 10, 2026 03:37

uranusjr reviewed Mar 10, 2026

View reviewed changes

ashb reviewed Mar 10, 2026

View reviewed changes

ashb approved these changes Mar 10, 2026

View reviewed changes

Conversation

dstandish commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xBis7 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ashb left a comment

Choose a reason for hiding this comment

Uh oh!

xBis7 commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dstandish commented Feb 27, 2026 •

edited

Loading

xBis7 left a comment •

edited

Loading

xBis7 commented Mar 10, 2026 •

edited

Loading